Home Explore Help
Register Sign In
J1X
/
huginn
Watch 2
Star 0
Fork 0

Browse Source
Merge branch 'master' into rss_agent_logs_exact_feed_on_error

Andrew Cantino 9 years ago
parent
438d09edf1 77ec02271f
commit
fe1e806368
22 changed files with 655 additions and 148 deletions
Show Diff Stats
  1. 1 1
      .env.example
  2. 7 0
      CHANGES.md
  3. 3 0
      Gemfile
  4. 1 4
      Gemfile.lock
  5. 4 2
      README.md
  6. 25 14
      app/concerns/dry_runnable.rb
  7. 161 0
      app/concerns/sortable_events.rb
  8. 1 0
      app/helpers/application_helper.rb
  9. 13 5
      app/models/agent.rb
  10. 6 2
      app/models/agents/data_output_agent.rb
  11. 34 15
      app/models/agents/rss_agent.rb
  12. 5 0
      app/models/agents/website_agent.rb
  13. 0 15
      app/models/scenario_import.rb
  14. 1 1
      config/initializers/delayed_job.rb
  15. 0 3
      lib/ar_mysql_column_charset.rb
  16. 0 85
      lib/prepend.rb
  17. 39 0
      lib/utils.rb
  18. 264 0
      spec/concerns/sortable_events_spec.rb
  19. 1 1
      spec/controllers/agents_controller_spec.rb
  20. 58 0
      spec/lib/utils_spec.rb
  21. 16 0
      spec/models/agents/data_output_agent_spec.rb
  22. 15 0
      spec/models/agents/rss_agent_spec.rb

+ 1 - 1
.env.example
View File 

@@ -167,7 +167,7 @@ TIMEZONE="Pacific Time (US & Canada)"
167 167 
 FAILED_JOBS_TO_KEEP=100
168 168
169 169 
 # Maximum runtime of background jobs in minutes
170 
-DELAYED_JOB_MAX_RUNTIME=20
170 
+DELAYED_JOB_MAX_RUNTIME=2
171 171
172 172 
 # Amount of seconds for delayed_job to sleep before checking for new jobs
173 173 
 DELAYED_JOB_SLEEP_DELAY=10

+ 7 - 0
CHANGES.md
View File 

@@ -1,12 +1,19 @@
1 1 
 # Changes
2 2
3 
+* Jul 30, 2015   - RssAgent can configure the order of events created via `events_order`.
4 
+* Jul 29, 2015   - WebsiteAgent can configure the order of events created via `events_order`.
5 
+* Jul 29, 2015   - DataOutputAgent can configure the order of events in the output via `events_order`.
3 6 
 * Jul 20, 2015   - Control Links (used by the SchedularAgent) are correctly exported in Scenarios.
4 7 
 * Jul 20, 2015   - keep\_events\_for was moved from days to seconds; Scenarios have a schema verison.
8 
+* Jul 8, 2015    - DataOutputAgent supports feed icon, and a new template variable `events`.
5 9 
 * Jul 1, 2015    - DeDuplicationAgent properly handles destruction of memory.
6 10 
 * Jun 26, 2015   - Add `max_events_per_run` to RssAgent.
7 11 
 * Jun 19, 2015   - Add `url_from_event` to WebsiteAgent.
8 12 
 * Jun 17, 2015   - RssAgent emits events for new feed items in chronological order.
13 
+* Jun 17, 2015   - Liquid filter `unescape` added.
14 
+* Jun 17, 2015   - Liquid filter `regex_replace` and `regex_replace_first` added, with escape sequence support.
9 15 
 * Jun 15, 2015   - Liquid filter `uri_expand` added.
16 
+* Jun 13, 2015   - Liquid templating engine is upgraded to version 3.
10 17 
 * Jun 12, 2015   - RSSAgent can now accept an array of URLs.
11 18 
 * Jun 8, 2015    - WebsiteAgent includes a `use_namespaces` option to enable XML namespaces.
12 19 
 * May 27, 2015   - Validation warns user if they have not provided a `path` when using JSONPath in WebsiteAgent.

+ 3 - 0
Gemfile
View File 

@@ -1,5 +1,8 @@
1 1 
 source 'https://rubygems.org'
2 2
3 
+# Ruby 2.0 is the minimum requirement
4 
+ruby ['2.0.0', RUBY_VERSION].max
5 
+
3 6 
 # Optional libraries.  To conserve RAM, comment out any that you don't need,
4 7 
 # then run `bundle` and commit the updated Gemfile and Gemfile.lock.
5 8 
 gem 'twilio-ruby', '~> 3.11.5'    # TwilioAgent

+ 1 - 4
Gemfile.lock
View File 

@@ -235,7 +235,7 @@ GEM
235 235 
     launchy (2.4.2)
236 236 
       addressable (~> 2.3)
237 237 
     libv8 (3.16.14.7)
238 
-    liquid (3.0.3)
238 
+    liquid (3.0.6)
239 239 
     listen (2.7.9)
240 240 
       celluloid (>= 0.15.2)
241 241 
       rb-fsevent (>= 0.9.3)
 
@@ -580,6 +580,3 @@ DEPENDENCIES
580 580 
   weibo_2!
581 581 
   wunderground (~> 1.2.0)
582 582 
   xmpp4r (~> 0.5.6)
583 
-
584 
-BUNDLED WITH
585 
-   1.10.5

+ 4 - 2
README.md
View File 

@@ -80,9 +80,11 @@ All agents have specs! Test all specs with `bundle exec rspec`, or test a specif
80 80
81 81 
 ## Deployment
82 82
83 
-[![Deploy](https://www.herokucdn.com/deploy/button.png)](https://heroku.com/deploy) (Takes a few minutes to setup.  Be sure to click 'View it' after launch!)
83 
+Try Huginn on Heroku: [![Deploy](https://www.herokucdn.com/deploy/button.png)](https://heroku.com/deploy) (Takes a few minutes to setup.  Be sure to click 'View it' after launch!)
84 84
85 
-Huginn can run on Heroku for free!  Please see [the Huginn Wiki](https://github.com/cantino/huginn/wiki#deploying-huginn) for detailed deployment strategies for different providers.
85 
+Huginn works on the free version of Heroku [with limitations](https://github.com/cantino/huginn/wiki/Run-Huginn-for-free-on-Heroku). For non-experimental use, we recommend Heroku's cheapest paid plan or our Docker container.
86 
+
87 
+Please see [the Huginn Wiki](https://github.com/cantino/huginn/wiki#deploying-huginn) for detailed deployment strategies for different providers.
86 88
87 89 
 ### Optional Setup
88 90

+ 25 - 14
app/concerns/dry_runnable.rb
View File 

@@ -1,10 +1,8 @@
1 1 
 module DryRunnable
2 
-  def dry_run!
3 
-    readonly!
2 
+  extend ActiveSupport::Concern
4 3
5 
-    class << self
6 
-      prepend Sandbox
7 
-    end
4 
+  def dry_run!
5 
+    @dry_run = true
8 6
9 7 
     log = StringIO.new
10 8 
     @dry_run_logger = Logger.new(log)
 
@@ -14,6 +12,7 @@ module DryRunnable
14 12
15 13 
     begin
16 14 
       raise "#{short_type} does not support dry-run" unless can_dry_run?
15 
+      readonly!
17 16 
       check
18 17 
     rescue => e
19 18 
       error "Exception during dry-run. #{e.message}: #{e.backtrace.join("\n")}"
 
@@ -23,28 +22,38 @@ module DryRunnable
23 22 
       memory: memory,
24 23 
       log: log.string,
25 24 
     )
25 
+  ensure
26 
+    @dry_run = false
26 27 
   end
27 28
28 29 
   def dry_run?
29 
-    is_a? Sandbox
30 
+    !!@dry_run
31 
+  end
32 
+
33 
+  included do
34 
+    prepend Wrapper
30 35 
   end
31 36
32 
-  module Sandbox
37 
+  module Wrapper
33 38 
     attr_accessor :results
34 39
35 40 
     def logger
41 
+      return super unless dry_run?
36 42 
       @dry_run_logger
37 43 
     end
38 44
39 
-    def save
40 
-      valid?
45 
+    def save(options = {})
46 
+      return super unless dry_run?
47 
+      perform_validations(options)
41 48 
     end
42 49
43 
-    def save!
44 
-      save or raise ActiveRecord::RecordNotSaved
50 
+    def save!(options = {})
51 
+      return super unless dry_run?
52 
+      save(options) or raise_record_invalid
45 53 
     end
46 54
47 55 
     def log(message, options = {})
56 
+      return super unless dry_run?
48 57 
       case options[:level] || 3
49 58 
       when 0..2
50 59 
         sev = Logger::DEBUG
 
@@ -57,10 +66,12 @@ module DryRunnable
57 66 
       logger.log(sev, message)
58 67 
     end
59 68
60 
-    def create_event(event_hash)
69 
+    def create_event(event)
70 
+      return super unless dry_run?
61 71 
       if can_create_events?
62 
-        @dry_run_results[:events] << event_hash[:payload]
63 
-        events.build({ user: user, expires_at: new_event_expiration_date }.merge(event_hash))
72 
+        event = build_event(event)
73 
+        @dry_run_results[:events] << event.payload
74 
+        event
64 75 
       else
65 76 
         error "This Agent cannot create events!"
66 77 
       end

+ 161 - 0
app/concerns/sortable_events.rb
View File 

@@ -0,0 +1,161 @@
1 
+module SortableEvents
2 
+  extend ActiveSupport::Concern
3 
+
4 
+  included do
5 
+    validate :validate_events_order
6 
+  end
7 
+
8 
+  def description_events_order(*args)
9 
+    self.class.description_events_order(*args)
10 
+  end
11 
+
12 
+  module ClassMethods
13 
+    def can_order_created_events!
14 
+      raise if cannot_create_events?
15 
+      prepend AutomaticSorter
16 
+    end
17 
+
18 
+    def can_order_created_events?
19 
+      include? AutomaticSorter
20 
+    end
21 
+
22 
+    def cannot_order_created_events?
23 
+      !can_order_created_events?
24 
+    end
25 
+
26 
+    def description_events_order(events = 'events created in each run')
27 
+      <<-MD.lstrip
28 
+        To specify the order of #{events}, set `events_order` to an array of sort keys, each of which looks like either `expression` or `[expression, type, descending]`, as described as follows:
29 
+
30 
+        * _expression_ is a Liquid template to generate a string to be used as sort key.
31 
+
32 
+        * _type_ (optional) is one of `string` (default), `number` and `time`, which specifies how to evaluate _expression_ for comparison.
33 
+
34 
+        * _descending_ (optional) is a boolean value to determine if comparison should be done in descending (reverse) order, which defaults to `false`.
35 
+
36 
+        Sort keys listed earlier take precedence over ones listed later.  For example, if you want to sort articles by the date and then by the author, specify `[["{{date}}", "time"], "{{author}}"]`.
37 
+
38 
+        Sorting is done stably, so even if all events have the same set of sort key values the original order is retained.  Also, a special Liquid variable `_index_` is provided, which contains the zero-based index number of each event, which means you can exactly reverse the order of events by specifying `[["{{_index_}}", "number", true]]`.
39 
+      MD
40 
+    end
41 
+  end
42 
+
43 
+  def can_order_created_events?
44 
+    self.class.can_order_created_events?
45 
+  end
46 
+
47 
+  def cannot_order_created_events?
48 
+    self.class.cannot_order_created_events?
49 
+  end
50 
+
51 
+  def events_order
52 
+    options['events_order']
53 
+  end
54 
+
55 
+  module AutomaticSorter
56 
+    def check
57 
+      return super unless events_order
58 
+      sorting_events do
59 
+        super
60 
+      end
61 
+    end
62 
+
63 
+    def receive(incoming_events)
64 
+      return super unless events_order
65 
+      # incoming events should be processed sequentially
66 
+      incoming_events.each do |event|
67 
+        sorting_events do
68 
+          super([event])
69 
+        end
70 
+      end
71 
+    end
72 
+
73 
+    def create_event(event)
74 
+      if @sortable_events
75 
+        event = build_event(event)
76 
+        @sortable_events << event
77 
+        event
78 
+      else
79 
+        super
80 
+      end
81 
+    end
82 
+
83 
+    private
84 
+
85 
+    def sorting_events(&block)
86 
+      @sortable_events = []
87 
+      yield
88 
+    ensure
89 
+      events, @sortable_events = @sortable_events, nil
90 
+      sort_events(events).each do |event|
91 
+        create_event(event)
92 
+      end
93 
+    end
94 
+  end
95 
+
96 
+  private
97 
+
98 
+  EXPRESSION_PARSER = {
99 
+    'string' => ->string { string },
100 
+    'number' => ->string { string.to_f },
101 
+    'time'   => ->string { Time.zone.parse(string) },
102 
+  }
103 
+  EXPRESSION_TYPES = EXPRESSION_PARSER.keys.freeze
104 
+
105 
+  def validate_events_order
106 
+    case order_by = events_order
107 
+    when nil
108 
+    when Array
109 
+      # Each tuple may be either [expression, type, desc] or just
110 
+      # expression.
111 
+      order_by.each do |expression, type, desc|
112 
+        case expression
113 
+        when String
114 
+          # ok
115 
+        else
116 
+          errors.add(:base, "first element of each events_order tuple must be a Liquid template")
117 
+          break
118 
+        end
119 
+        case type
120 
+        when nil, *EXPRESSION_TYPES
121 
+          # ok
122 
+        else
123 
+          errors.add(:base, "second element of each events_order tuple must be #{EXPRESSION_TYPES.to_sentence(last_word_connector: ' or ')}")
124 
+          break
125 
+        end
126 
+        if !desc.nil? && boolify(desc).nil?
127 
+          errors.add(:base, "third element of each events_order tuple must be a boolean value")
128 
+          break
129 
+        end
130 
+      end
131 
+    else
132 
+      errors.add(:base, "events_order must be an array of arrays")
133 
+    end
134 
+  end
135 
+
136 
+  # Sort given events in order specified by the "events_order" option
137 
+  def sort_events(events)
138 
+    order_by = events_order.presence or
139 
+      return events
140 
+
141 
+    orders = order_by.map { |_, _, desc = false| boolify(desc) }
142 
+
143 
+    Utils.sort_tuples!(
144 
+      events.map.with_index { |event, index|
145 
+        interpolate_with(event) {
146 
+          interpolation_context['_index_'] = index
147 
+          order_by.map { |expression, type, _|
148 
+            string = interpolate_string(expression)
149 
+            begin
150 
+              EXPRESSION_PARSER[type || 'string'.freeze][string]
151 
+            rescue
152 
+              error "Cannot parse #{string.inspect} as #{type}; treating it as string"
153 
+              string
154 
+            end
155 
+          }
156 
+        } << index << event  # index is to make sorting stable
157 
+      },
158 
+      orders
159 
+    ).collect!(&:last)
160 
+  end
161 
+end

+ 1 - 0
app/helpers/application_helper.rb
View File 

@@ -80,6 +80,7 @@ module ApplicationHelper
80 80 
   end
81 81
82 82 
   def service_label(service)
83 
+    return if service.nil?
83 84 
     content_tag :span, [
84 85 
       omniauth_provider_icon(service.provider),
85 86 
       service_label_text(service)

+ 13 - 5
app/models/agent.rb
View File 

@@ -13,6 +13,7 @@ class Agent < ActiveRecord::Base
13 13 
   include HasGuid
14 14 
   include LiquidDroppable
15 15 
   include DryRunnable
16 
+  include SortableEvents
16 17
17 18 
   markdown_class_attributes :description, :event_description
18 19 
 
@@ -104,12 +105,19 @@ class Agent < ActiveRecord::Base
104 105 
     raise "Implement me in your subclass"
105 106 
   end
106 107
107 
-  def create_event(attrs)
108 
+  def build_event(event)
109 
+    event = events.build(event) if event.is_a?(Hash)
110 
+    event.agent = self
111 
+    event.user = user
112 
+    event.expires_at ||= new_event_expiration_date
113 
+    event
114 
+  end
115 
+
116 
+  def create_event(event)
108 117 
     if can_create_events?
109 
-      events.create!({
110 
-         :user => user,
111 
-         :expires_at => new_event_expiration_date
112 
-      }.merge(attrs))
118 
+      event = build_event(event)
119 
+      event.save!
120 
+      event
113 121 
     else
114 122 
       error "This Agent cannot create events!"
115 123 
     end

+ 6 - 2
app/models/agents/data_output_agent.rb
View File 

@@ -40,11 +40,15 @@ module Agents
40 40 
               "_contents": "tag contents (can be an object for nesting)"
41 41 
             }
42 42
43 
+        # Ordering events in the output
44 
+
45 
+        #{description_events_order('events in the output')}
46 
+
43 47 
         # Liquid Templating
44 48
45 49 
         In Liquid templating, the following variable is available:
46 50
47 
-        * `events`: An array of events being output, sorted in descending order up to `events_to_show` in number.  For example, if source events contain a site title in the `site_title` key, you can refer to it in `template.title` by putting `{{events.first.site_title}}`.
51 
+        * `events`: An array of events being output, sorted in the given order, up to `events_to_show` in number.  For example, if source events contain a site title in the `site_title` key, you can refer to it in `template.title` by putting `{{events.first.site_title}}`.
48 52
49 53 
       MD
50 54 
     end
 
@@ -134,7 +138,7 @@ module Agents
134 138 
         end
135 139 
       end
136 140
137 
-      source_events = received_events.order(id: :desc).limit(events_to_show).to_a
141 
+      source_events = sort_events(received_events.order(id: :desc).limit(events_to_show).to_a)
138 142
139 143 
       interpolation_context.stack do
140 144 
         interpolation_context['events'] = source_events

+ 34 - 15
app/models/agents/rss_agent.rb
View File 

@@ -9,6 +9,8 @@ module Agents
9 9 
     can_dry_run!
10 10 
     default_schedule "every_1d"
11 11
12 
+    DEFAULT_EVENTS_ORDER = [['{{date_published}}', 'time'], ['{{last_updated}}', 'time']]
13 
+
12 14 
     description do
13 15 
       <<-MD
14 16 
         This Agent consumes RSS feeds and emits events when they change.
 
@@ -29,6 +31,12 @@ module Agents
29 31 
           * `disable_url_encoding` - Set to `true` to disable url encoding.
30 32 
           * `user_agent` - A custom User-Agent name (default: "Faraday v#{Faraday::VERSION}").
31 33 
           * `max_events_per_run` - Limit number of events created (items parsed) per run for feed.
34 
+
35 
+        # Ordering Events
36 
+
37 
+        #{description_events_order}
38 
+
39 
+        In this Agent, the default value for `events_order` is `#{DEFAULT_EVENTS_ORDER.to_json}`.
32 40 
       MD
33 41 
     end
34 42 
 
@@ -70,6 +78,11 @@ module Agents
70 78 
       end
71 79
72 80 
       validate_web_request_options!
81 
+      validate_events_order
82 
+    end
83 
+
84 
+    def events_order
85 
+      super.presence || DEFAULT_EVENTS_ORDER
73 86 
     end
74 87
75 88 
     def check
 
@@ -84,26 +97,15 @@ module Agents
84 97 
       response = faraday.get(url)
85 98 
       if response.success?
86 99 
         feed = FeedNormalizer::FeedNormalizer.parse(response.body)
87 
-        feed.clean! if interpolated['clean'] == 'true'
100 
+        feed.clean! if boolify(interpolated['clean'])
88 101 
         max_events = (interpolated['max_events_per_run'].presence || 0).to_i
89 102 
         created_event_count = 0
90 
-        feed.entries.sort_by { |entry| [entry.date_published, entry.last_updated] }.each.with_index do |entry, index|
103 
+        sort_events(feed_to_events(feed)).each.with_index do |event, index|
91 104 
           break if max_events && max_events > 0 && index >= max_events
92 
-          entry_id = get_entry_id(entry)
105 
+          entry_id = event.payload[:id]
93 106 
           if check_and_track(entry_id)
94 107 
             created_event_count += 1
95 
-            create_event(payload: {
96 
-                           id: entry_id,
97 
-                           date_published: entry.date_published,
98 
-                           last_updated: entry.last_updated,
99 
-                           url: entry.url,
100 
-                           urls: entry.urls,
101 
-                           description: entry.description,
102 
-                           content: entry.content,
103 
-                           title: entry.title,
104 
-                           authors: entry.authors,
105 
-                           categories: entry.categories
106 
-                         })
108 
+            create_event(event)
107 109 
           end
108 110 
         end
109 111 
         log "Fetched #{url} and created #{created_event_count} event(s)."
 
@@ -128,5 +130,22 @@ module Agents
128 130 
         true
129 131 
       end
130 132 
     end
133 
+
134 
+    def feed_to_events(feed)
135 
+      feed.entries.map { |entry|
136 
+        Event.new(payload: {
137 
+                    id: get_entry_id(entry),
138 
+                    date_published: entry.date_published,
139 
+                    last_updated: entry.last_updated,
140 
+                    url: entry.url,
141 
+                    urls: entry.urls,
142 
+                    description: entry.description,
143 
+                    content: entry.content,
144 
+                    title: entry.title,
145 
+                    authors: entry.authors,
146 
+                    categories: entry.categories
147 
+                  })
148 
+      }
149 
+    end
131 150 
   end
132 151 
 end

+ 5 - 0
app/models/agents/website_agent.rb
View File 

@@ -6,6 +6,7 @@ module Agents
6 6 
     include WebRequestConcern
7 7
8 8 
     can_dry_run!
9 
+    can_order_created_events!
9 10
10 11 
     default_schedule "every_12h"
11 12 
 
@@ -105,6 +106,10 @@ module Agents
105 106 
           * `status`: HTTP status as integer. (Almost always 200)
106 107
107 108 
           * `headers`: Response headers; for example, `{{ _response_.headers.Content-Type }}` expands to the value of the Content-Type header.  Keys are insensitive to cases and -/_.
109 
+
110 
+      # Ordering Events
111 
+
112 
+      #{description_events_order}
108 113 
     MD
109 114
110 115 
     event_description do

+ 0 - 15
app/models/scenario_import.rb
View File 

@@ -278,21 +278,6 @@ class ScenarioImport
278 278 
       yield 'disabled', disabled, boolean if disabled.requires_merge?
279 279 
     end
280 280
281 
-    # Unfortunately Ruby 1.9's OpenStruct doesn't expose [] and []=.
282 
-    unless instance_methods.include?(:[]=)
283 
-      def [](key)
284 
-        self.send(sanitize key)
285 
-      end
286 
-
287 
-      def []=(key, val)
288 
-        self.send("#{sanitize key}=", val)
289 
-      end
290 
-
291 
-      def sanitize(key)
292 
-        key.gsub(/[^a-zA-Z0-9_-]/, '')
293 
-      end
294 
-    end
295 
-
296 281 
     def agent_instance
297 282 
       "Agents::#{self.type.updated}".constantize.new
298 283 
     end

+ 1 - 1
config/initializers/delayed_job.rb
View File 

@@ -1,6 +1,6 @@
1 1 
 Delayed::Worker.destroy_failed_jobs = false
2 2 
 Delayed::Worker.max_attempts = 5
3 
-Delayed::Worker.max_run_time = (ENV['DELAYED_JOB_MAX_RUNTIME'].presence || 20).to_i.minutes
3 
+Delayed::Worker.max_run_time = (ENV['DELAYED_JOB_MAX_RUNTIME'].presence || 2).to_i.minutes
4 4 
 Delayed::Worker.read_ahead = 5
5 5 
 Delayed::Worker.default_priority = 10
6 6 
 Delayed::Worker.delay_jobs = !Rails.env.test?

+ 0 - 3
lib/ar_mysql_column_charset.rb
View File 

@@ -1,6 +1,3 @@
1 
-# Module#prepend support for Ruby 1.9
2 
-require 'prepend' unless Module.method_defined?(:prepend)
3 
-
4 1 
 require 'active_support'
5 2
6 3 
 ActiveSupport.on_load :active_record do

+ 0 - 85
lib/prepend.rb
View File 

@@ -1,85 +0,0 @@
1 
-# Fake implementation of prepend(), which does not support overriding
2 
-# inherited methods nor methods that are formerly overridden by
3 
-# another invocation of prepend().
4 
-#
5 
-# Here's what <Original>.prepend(<Wrapper>) does:
6 
-#
7 
-# - Create an anonymous stub module (hereinafter <Stub>) and define
8 
-#   <Stub>#<method> that calls #<method>_without_<Wrapper> for each
9 
-#   instance method of <Wrapper>.
10 
-#
11 
-# - Rename <Original>#<method> to #<method>_without_<Wrapper> for each
12 
-#   instance method of <Wrapper>.
13 
-#
14 
-# - Include <Stub> and <Wrapper> into <Original> in that order.
15 
-#
16 
-# This way, a call of <Original>#<method> is dispatched to
17 
-# <Wrapper><method>, which may call super which is dispatched to
18 
-# <Stub>#<method>, which finally calls
19 
-# <Original>#<method>_without_<Wrapper> which is used to be called
20 
-# <Original>#<method>.
21 
-#
22 
-# Usage:
23 
-#
24 
-#     class Mechanize
25 
-#       # module with methods that overrides those of X
26 
-#       module Y
27 
-#       end
28 
-#
29 
-#       unless X.respond_to?(:prepend, true)
30 
-#         require 'mechanize/prependable'
31 
-#         X.extend(Prependable)
32 
-#       end
33 
-#
34 
-#       class X
35 
-#         prepend Y
36 
-#       end
37 
-#     end
38 
-class Module
39 
-  def prepend(mod)
40 
-    stub = Module.new
41 
-
42 
-    mod_id = (mod.name || 'Module__%d' % mod.object_id).gsub(/::/, '__')
43 
-
44 
-    mod.instance_methods.each { |name|
45 
-      method_defined?(name) or next
46 
-
47 
-      original = instance_method(name)
48 
-      next if original.owner != self
49 
-
50 
-      name = name.to_s
51 
-      name_without = name.sub(/(?=[?!=]?\z)/) { '_without_%s' % mod_id }
52 
-
53 
-      arity = original.arity
54 
-      arglist = (
55 
-        if arity >= 0
56 
-          (1..arity).map { |i| 'x%d' % i }
57 
-        else
58 
-          (1..(-arity - 1)).map { |i| 'x%d' % i } << '*a'
59 
-        end << '&b'
60 
-      ).join(', ')
61 
-
62 
-      if name.end_with?('=')
63 
-        stub.module_eval %{
64 
-          def #{name}(#{arglist})
65 
-            __send__(:#{name_without}, #{arglist})
66 
-          end
67 
-        }
68 
-      else
69 
-        stub.module_eval %{
70 
-          def #{name}(#{arglist})
71 
-            #{name_without}(#{arglist})
72 
-          end
73 
-        }
74 
-      end
75 
-      module_eval {
76 
-        alias_method name_without, name
77 
-        remove_method name
78 
-      }
79 
-    }
80 
-
81 
-    include stub
82 
-    include mod
83 
-  end
84 
-  private :prepend
85 
-end unless Module.method_defined?(:prepend)

+ 39 - 0
lib/utils.rb
View File 

@@ -79,4 +79,43 @@ module Utils
79 79 
   def self.pretty_jsonify(thing)
80 80 
     JSON.pretty_generate(thing).gsub('</', '<\/')
81 81 
   end
82 
+
83 
+  class TupleSorter
84 
+    class SortableTuple
85 
+      attr_reader :array
86 
+
87 
+      # The <=> method will call orders[n] to determine if the nth element
88 
+      # should be compared in descending order.
89 
+      def initialize(array, orders = [])
90 
+        @array = array
91 
+        @orders = orders
92 
+      end
93 
+
94 
+      def <=> other
95 
+        other = other.array
96 
+        @array.each_with_index do |e, i|
97 
+          o = other[i]
98 
+          case cmp = e <=> o || e.to_s <=> o.to_s
99 
+          when 0
100 
+            next
101 
+          else
102 
+            return @orders[i] ? -cmp : cmp
103 
+          end
104 
+        end
105 
+        0
106 
+      end
107 
+    end
108 
+
109 
+    class << self
110 
+      def sort!(array, orders = [])
111 
+        array.sort_by! do |e|
112 
+          SortableTuple.new(e, orders)
113 
+        end
114 
+      end
115 
+    end
116 
+  end
117 
+
118 
+  def self.sort_tuples!(array, orders = [])
119 
+    TupleSorter.sort!(array, orders)
120 
+  end
82 121 
 end

+ 264 - 0
spec/concerns/sortable_events_spec.rb
View File 

@@ -0,0 +1,264 @@
1 
+require 'spec_helper'
2 
+
3 
+describe SortableEvents do
4 
+  let(:agent_class) {
5 
+    Class.new(Agent) do
6 
+      include SortableEvents
7 
+
8 
+      default_schedule 'never'
9 
+
10 
+      def self.valid_type?(name)
11 
+        true
12 
+      end
13 
+    end
14 
+  }
15 
+
16 
+  def new_agent(events_order = nil)
17 
+    options = {}
18 
+    options['events_order'] = events_order if events_order
19 
+    agent_class.new(name: 'test', options: options) { |agent|
20 
+      agent.user = users(:bob)
21 
+    }
22 
+  end
23 
+
24 
+  describe 'validations' do
25 
+    let(:agent_class) {
26 
+      Class.new(Agent) do
27 
+        include SortableEvents
28 
+
29 
+        default_schedule 'never'
30 
+
31 
+        def self.valid_type?(name)
32 
+          true
33 
+        end
34 
+      end
35 
+    }
36 
+
37 
+    def new_agent(events_order = nil)
38 
+      options = {}
39 
+      options['events_order'] = events_order if events_order
40 
+      agent_class.new(name: 'test', options: options) { |agent|
41 
+        agent.user = users(:bob)
42 
+      }
43 
+    end
44 
+
45 
+    it 'should allow events_order to be unspecified, null or an empty array' do
46 
+      expect(new_agent()).to be_valid
47 
+      expect(new_agent(nil)).to be_valid
48 
+      expect(new_agent([])).to be_valid
49 
+    end
50 
+
51 
+    it 'should not allow events_order to be a non-array object' do
52 
+      agent = new_agent(0)
53 
+      expect(agent).not_to be_valid
54 
+      expect(agent.errors[:base]).to include(/events_order/)
55 
+
56 
+      agent = new_agent('')
57 
+      expect(agent).not_to be_valid
58 
+      expect(agent.errors[:base]).to include(/events_order/)
59 
+
60 
+      agent = new_agent({})
61 
+      expect(agent).not_to be_valid
62 
+      expect(agent.errors[:base]).to include(/events_order/)
63 
+    end
64 
+
65 
+    it 'should not allow events_order to be an array containing unexpected objects' do
66 
+      agent = new_agent(['{{key}}', 1])
67 
+      expect(agent).not_to be_valid
68 
+      expect(agent.errors[:base]).to include(/events_order/)
69 
+
70 
+      agent = new_agent(['{{key1}}', ['{{key2}}', 'unknown']])
71 
+      expect(agent).not_to be_valid
72 
+      expect(agent.errors[:base]).to include(/events_order/)
73 
+    end
74 
+
75 
+    it 'should allow events_order to be an array containing strings and valid tuples' do
76 
+      agent = new_agent(['{{key1}}', ['{{key2}}'], ['{{key3}}', 'number']])
77 
+      expect(agent).to be_valid
78 
+
79 
+      agent = new_agent(['{{key1}}', ['{{key2}}'], ['{{key3}}', 'number'], ['{{key4}}', 'time', true]])
80 
+      expect(agent).to be_valid
81 
+    end
82 
+  end
83 
+
84 
+  describe 'sort_events' do
85 
+    let(:payloads) {
86 
+      [
87 
+        { 'title' => 'TitleA', 'score' => 4,  'updated_on' => '7 Jul 2015' },
88 
+        { 'title' => 'TitleB', 'score' => 2,  'updated_on' => '25 Jun 2014' },
89 
+        { 'title' => 'TitleD', 'score' => 10, 'updated_on' => '10 Jan 2015' },
90 
+        { 'title' => 'TitleC', 'score' => 10, 'updated_on' => '9 Feb 2015' },
91 
+      ]
92 
+    }
93 
+
94 
+    let(:events) {
95 
+      payloads.map { |payload| Event.new(payload: payload) }
96 
+    }
97 
+
98 
+    it 'should sort events by a given key' do
99 
+      agent = new_agent(['{{title}}'])
100 
+      expect(agent.__send__(:sort_events, events).map { |e| e.payload['title'] }).to eq(%w[TitleA TitleB TitleC TitleD])
101 
+
102 
+      agent = new_agent([['{{title}}', 'string', true]])
103 
+      expect(agent.__send__(:sort_events, events).map { |e| e.payload['title'] }).to eq(%w[TitleD TitleC TitleB TitleA])
104 
+    end
105 
+
106 
+    it 'should sort events by multiple keys' do
107 
+      agent = new_agent([['{{score}}', 'number'], '{{title}}'])
108 
+      expect(agent.__send__(:sort_events, events).map { |e| e.payload['title'] }).to eq(%w[TitleB TitleA TitleC TitleD])
109 
+
110 
+      agent = new_agent([['{{score}}', 'number'], ['{{title}}', 'string', true]])
111 
+      expect(agent.__send__(:sort_events, events).map { |e| e.payload['title'] }).to eq(%w[TitleB TitleA TitleD TitleC])
112 
+    end
113 
+
114 
+    it 'should sort events by time' do
115 
+      agent = new_agent([['{{updated_on}}', 'time']])
116 
+      expect(agent.__send__(:sort_events, events).map { |e| e.payload['title'] }).to eq(%w[TitleB TitleD TitleC TitleA])
117 
+    end
118 
+
119 
+    it 'should sort events stably' do
120 
+      agent = new_agent(['<constant>'])
121 
+      expect(agent.__send__(:sort_events, events).map { |e| e.payload['title'] }).to eq(%w[TitleA TitleB TitleD TitleC])
122 
+
123 
+      agent = new_agent([['<constant>', 'string', true]])
124 
+      expect(agent.__send__(:sort_events, events).map { |e| e.payload['title'] }).to eq(%w[TitleA TitleB TitleD TitleC])
125 
+    end
126 
+
127 
+    it 'should support _index_' do
128 
+      agent = new_agent([['{{_index_}}', 'number', true]])
129 
+      expect(agent.__send__(:sort_events, events).map { |e| e.payload['title'] }).to eq(%w[TitleC TitleD TitleB TitleA])
130 
+    end
131 
+  end
132 
+
133 
+  describe 'automatic event sorter' do
134 
+    describe 'declaration' do
135 
+      let(:passive_agent_class) {
136 
+        Class.new(Agent) do
137 
+          include SortableEvents
138 
+
139 
+          cannot_create_events!
140 
+        end
141 
+      }
142 
+
143 
+      let(:active_agent_class) {
144 
+        Class.new(Agent) do
145 
+          include SortableEvents
146 
+        end
147 
+      }
148 
+
149 
+      describe 'can_order_created_events!' do
150 
+        it 'should refuse to work if called from an Agent that cannot create events' do
151 
+          expect {
152 
+            passive_agent_class.class_eval do
153 
+              can_order_created_events!
154 
+            end
155 
+          }.to raise_error
156 
+        end
157 
+
158 
+        it 'should work if called from an Agent that can create events' do
159 
+          expect {
160 
+            active_agent_class.class_eval do
161 
+              can_order_created_events!
162 
+            end
163 
+          }.not_to raise_error
164 
+        end
165 
+      end
166 
+
167 
+      describe 'can_order_created_events?' do
168 
+        it 'should return false unless an Agent declares can_order_created_events!' do
169 
+          expect(active_agent_class.can_order_created_events?).to eq(false)
170 
+          expect(active_agent_class.new.can_order_created_events?).to eq(false)
171 
+        end
172 
+
173 
+        it 'should return true if an Agent declares can_order_created_events!' do
174 
+          active_agent_class.class_eval do
175 
+            can_order_created_events!
176 
+          end
177 
+
178 
+          expect(active_agent_class.can_order_created_events?).to eq(true)
179 
+          expect(active_agent_class.new.can_order_created_events?).to eq(true)
180 
+        end
181 
+      end
182 
+    end
183 
+
184 
+    describe 'behavior' do
185 
+      class Agents::EventOrderableAgent < Agent
186 
+        include SortableEvents
187 
+
188 
+        default_schedule 'never'
189 
+
190 
+        can_order_created_events!
191 
+
192 
+        attr_accessor :payloads_to_emit
193 
+
194 
+        def self.valid_type?(name)
195 
+          true
196 
+        end
197 
+
198 
+        def check
199 
+          payloads_to_emit.each do |payload|
200 
+            create_event payload: payload
201 
+          end
202 
+        end
203 
+
204 
+        def receive(events)
205 
+          events.each do |event|
206 
+            payloads_to_emit.each do |payload|
207 
+              create_event payload: payload.merge('title' => payload['title'] + event.payload['title_suffix'])
208 
+            end
209 
+          end
210 
+        end
211 
+      end
212 
+
213 
+      def new_agent(events_order = nil)
214 
+        options = {}
215 
+        options['events_order'] = events_order if events_order
216 
+        Agents::EventOrderableAgent.new(name: 'test', options: options) { |agent|
217 
+          agent.user = users(:bob)
218 
+          agent.payloads_to_emit = payloads
219 
+        }
220 
+      end
221 
+
222 
+      let(:payloads) {
223 
+        [
224 
+          { 'title' => 'TitleA', 'score' => 4,  'updated_on' => '7 Jul 2015' },
225 
+          { 'title' => 'TitleB', 'score' => 2,  'updated_on' => '25 Jun 2014' },
226 
+          { 'title' => 'TitleD', 'score' => 10, 'updated_on' => '10 Jan 2015' },
227 
+          { 'title' => 'TitleC', 'score' => 10, 'updated_on' => '9 Feb 2015' },
228 
+        ]
229 
+      }
230 
+
231 
+      it 'should keep the order of created events unless events_order is specified' do
232 
+        [[], [nil], [[]]].each do |args|
233 
+          agent = new_agent(*args)
234 
+          agent.save!
235 
+          expect { agent.check }.to change { Event.count }.by(4)
236 
+          events = agent.events.last(4).sort_by(&:id)
237 
+          expect(events.map { |event| event.payload['title'] }).to eq(%w[TitleA TitleB TitleD TitleC])
238 
+        end
239 
+      end
240 
+
241 
+      it 'should sort events created in check() in the order specified in events_order' do
242 
+        agent = new_agent([['{{score}}', 'number'], ['{{title}}', 'string', true]])
243 
+        agent.save!
244 
+        expect { agent.check }.to change { Event.count }.by(4)
245 
+        events = agent.events.last(4).sort_by(&:id)
246 
+        expect(events.map { |event| event.payload['title'] }).to eq(%w[TitleB TitleA TitleD TitleC])
247 
+      end
248 
+
249 
+      it 'should sort events created in receive() in the order specified in events_order' do
250 
+        agent = new_agent([['{{score}}', 'number'], ['{{title}}', 'string', true]])
251 
+        agent.save!
252 
+        expect {
253 
+          agent.receive([Event.new(payload: { 'title_suffix' => ' [new]' }),
254 
+                         Event.new(payload: { 'title_suffix' => ' [popular]' })])
255 
+        }.to change { Event.count }.by(8)
256 
+        events = agent.events.last(8).sort_by(&:id)
257 
+        expect(events.map { |event| event.payload['title'] }).to eq([
258 
+          'TitleB [new]',     'TitleA [new]',     'TitleD [new]',     'TitleC [new]',
259 
+          'TitleB [popular]', 'TitleA [popular]', 'TitleD [popular]', 'TitleC [popular]',
260 
+        ])
261 
+      end
262 
+    end
263 
+  end
264 
+end

+ 1 - 1
spec/controllers/agents_controller_spec.rb
View File 

@@ -372,7 +372,7 @@ describe AgentsController do
372 372 
       sign_in users(:bob)
373 373 
       agent = agents(:bob_weather_agent)
374 374 
       expect {
375 
-        post :dry_run, id: agents(:bob_website_agent), agent: valid_attributes(name: 'New Name')
375 
+        post :dry_run, id: agent, agent: valid_attributes(name: 'New Name')
376 376 
       }.not_to change {
377 377 
         [users(:bob).agents.count, users(:bob).events.count, users(:bob).logs.count, agent.name, agent.updated_at]
378 378 
       }

+ 58 - 0
spec/lib/utils_spec.rb
View File 

@@ -114,4 +114,62 @@ describe Utils do
114 114 
       expect(cleaned_json).to include("<\\/script>")
115 115 
     end
116 116 
   end
117 
+
118 
+  describe "#sort_tuples!" do
119 
+    let(:tuples) {
120 
+      time = Time.now
121 
+      [
122 
+        [2, "a", time - 1],  # 0
123 
+        [2, "b", time - 1],  # 1
124 
+        [1, "b", time - 1],  # 2
125 
+        [1, "b", time],      # 3
126 
+        [1, "a", time],      # 4
127 
+        [2, "a", time + 1],  # 5
128 
+        [2, "a", time],      # 6
129 
+      ]
130 
+    }
131 
+
132 
+    it "sorts tuples like arrays by default" do
133 
+      expected = tuples.values_at(4, 2, 3, 0, 6, 5, 1)
134 
+
135 
+      Utils.sort_tuples!(tuples)
136 
+      expect(tuples).to eq expected
137 
+    end
138 
+
139 
+    it "sorts tuples in order specified: case 1" do
140 
+      # order by x1 asc, x2 desc, c3 asc
141 
+      orders = [false, true, false]
142 
+      expected = tuples.values_at(2, 3, 4, 1, 0, 6, 5)
143 
+
144 
+      Utils.sort_tuples!(tuples, orders)
145 
+      expect(tuples).to eq expected
146 
+    end
147 
+
148 
+    it "sorts tuples in order specified: case 2" do
149 
+      # order by x1 desc, x2 asc, c3 desc
150 
+      orders = [true, false, true]
151 
+      expected = tuples.values_at(5, 6, 0, 1, 4, 3, 2)
152 
+
153 
+      Utils.sort_tuples!(tuples, orders)
154 
+      expect(tuples).to eq expected
155 
+    end
156 
+
157 
+    it "always succeeds in sorting even if it finds pairs of incomparable objects" do
158 
+      time = Time.now
159 
+      tuples = [
160 
+        [2,   "a", time - 1],  # 0
161 
+        [1,   "b", nil],       # 1
162 
+        [1,   "b", time],      # 2
163 
+        ["2", nil, time],      # 3
164 
+        [1,   nil, time],      # 4
165 
+        [nil, "a", time + 1],  # 5
166 
+        [2,   "a", time],      # 6
167 
+      ]
168 
+      orders = [true, false, true]
169 
+      expected = tuples.values_at(3, 6, 0, 4, 2, 1, 5)
170 
+
171 
+      Utils.sort_tuples!(tuples, orders)
172 
+      expect(tuples).to eq expected
173 
+    end
174 
+  end
117 175 
 end

+ 16 - 0
spec/models/agents/data_output_agent_spec.rb
View File 

@@ -209,6 +209,22 @@ describe Agents::DataOutputAgent do
209 209 
         })
210 210 
       end
211 211
212 
+      describe 'ordering' do
213 
+        before do
214 
+          agent.options['events_order'] = ['{{title}}']
215 
+        end
216 
+
217 
+        it 'can reorder the events_to_show last events based on a Liquid expression' do
218 
+          asc_content, _status, _content_type = agent.receive_web_request({ 'secret' => 'secret2' }, 'get', 'application/json')
219 
+          expect(asc_content['items'].map {|i| i["title"] }).to eq(["Evolving", "Evolving again", "Evolving yet again with a past date"])
220 
+
221 
+          agent.options['events_order'] = [['{{title}}', 'string', true]]
222 
+
223 
+          desc_content, _status, _content_type = agent.receive_web_request({ 'secret' => 'secret2' }, 'get', 'application/json')
224 
+          expect(desc_content['items']).to eq(asc_content['items'].reverse)
225 
+        end
226 
+      end
227 
+
212 228 
       describe "interpolating \"events\"" do
213 229 
         before do
214 230 
           agent.options['template']['title'] = "XKCD comics as a feed{% if events.first.site_title %} ({{events.first.site_title}}){% endif %}"

+ 15 - 0
spec/models/agents/rss_agent_spec.rb
View File 

@@ -66,6 +66,21 @@ describe Agents::RssAgent do
66 66 
       expect(last.payload['urls']).to eq(["https://github.com/cantino/huginn/commit/d465158f77dcd9078697e6167b50abbfdfa8b1af"])
67 67 
     end
68 68
69 
+    it "should emit items as events in the order specified in the events_order option" do
70 
+      expect {
71 
+        agent.options['events_order'] = ['{{title | replace_regex: "^[[:space:]]+", "" }}']
72 
+        agent.check
73 
+      }.to change { agent.events.count }.by(20)
74 
+
75 
+      first, *, last = agent.events.last(20)
76 
+      expect(first.payload['title'].strip).to eq('upgrade rails and gems')
77 
+      expect(first.payload['url']).to eq("https://github.com/cantino/huginn/commit/87a7abda23a82305d7050ac0bb400ce36c863d01")
78 
+      expect(first.payload['urls']).to eq(["https://github.com/cantino/huginn/commit/87a7abda23a82305d7050ac0bb400ce36c863d01"])
79 
+      expect(last.payload['title'].strip).to eq('Dashed line in a diagram indicates propagate_immediately being false.')
80 
+      expect(last.payload['url']).to eq("https://github.com/cantino/huginn/commit/0e80f5341587aace2c023b06eb9265b776ac4535")
81 
+      expect(last.payload['urls']).to eq(["https://github.com/cantino/huginn/commit/0e80f5341587aace2c023b06eb9265b776ac4535"])
82 
+    end
83 
+
69 84 
     it "should track ids and not re-emit the same item when seen again" do
70 85 
       agent.check
71 86 
       expect(agent.memory['seen_ids']).to eq(agent.events.map {|e| e.payload['id'] })